Skip to main content

Incident Report: WBTC/USD Price Deviation Alert on Zircuit

Date: 2024-12-18
Time: 23:21:06 UTC
Duration: 30 minutes

Description​

A price deviation alert was triggered for WBTC/USD on Zircuit. The deviation was detected by Airseeker and resolved automatically after three update cycles.

Root Cause​

The incident appears to have been caused by a race condition where the signed API returned slightly different deviation values to Airseeker (<1%) and Centurion (>1%) at approximately the same time. The signed data deviation started at 23:19 with an approximately 1% deviation.

Impact​

  • Brief deviation alert triggered
  • No service disruption
  • System worked as designed with automatic recovery
  • No user impact as price updates continued to process normally

Timeline​

  • 23:19:00 - Initial signed data deviation began (~1%)
  • 23:21:06 - Airseeker first detected the deviation
  • 23:21:22 - First provider confirmed update
  • 23:21:26 - Second provider confirmed update
  • 23:21:31 - Third provider confirmed update, incident resolved

Lessons Learned​

  1. The current 1% threshold may be too sensitive, leading to occasional false positives in edge cases
  2. The system's automatic recovery mechanisms worked as intended
  3. The difference between Airseeker and Centurion deviation detection thresholds can create edge cases

Actions Taken​

  • Immediate: No immediate action required as system self-corrected
  • Proposed Future Actions:
  • Consider adding a small tolerance buffer (e.g., 1.05% instead of 1%) to Centurion's deviation threshold
  • Add to deviation alert handling protocol to wait another run cycle if deviation is under 1.05%
  • Create issue in historyWorker alert actions for handling borderline deviations

Incident Reviewer(s)​

  • Arda
  • Burak
  • Warren
Last updated on by Warren Anderson